Advanced Machine Learning Practical 3 : Classification ( SVM , RVM & AdaBoost ) Professor : Aude Billard

During this week’s practical we will focus on understanding and comparing the performance of the different classification methods seen in class, namely SVM and its variants (C-SVM and ν-SVM, RVM) and an instance of a Boosting method, specifically AdaBoost. In non-linear classification methods such as SVM/RVM, we seek to find the optimal hyper-parameters (C: penalty, ν: bounds, σ: kernel width for RBF) which will optimize the objective function (and consequently the parameters of the class decision function). Choosing the best hyper-parameters for your dataset is not a trivial task, we will analyze their effect on the classifier and how to choose an admissible range of parameters. A standard way of finding these optimal hyper-parameters is by doing a grid search, i.e. systematically evaluating each possible combination of parameters within a given range. We will do this for different datasets and discuss the difference in performance, model complexity and sensitivity to hyper-parameters. On the other hand, Boosting chooses weak classifiers (WC) iteratively among a large set of randomly created WC and combines them to create a strong classifier, by increasing the weights of datapoints not well classified by the previous combination of WC. We will compare the performance of SVM vs. Adaboost (with decision stumps as the WC) on multiple datasets. We will also evaluate which method is more reliable when handling noisy data, data with outliers and data with unbalanced classes.